Understanding the Open Directory Project through Formal Concept Analysis

Directories may contain certain abnormalities in their structure certain categories may contain unrelated sub-categories, whilst other categories may have integral subcategories that are absent. By understanding these irregularities, the effectiveness and efficiency of information management can be greatly improved. The Open Directory Project (ODP) is one such directory that aims to categorise the Internet. The ODP is an ambitious project which spans over 500,000 categories with categories ranging from archery to zebras. However, the uniqueness of the directory arises from its editorial structure. With over 57,000 editors, and each given authority to moderate a section of the directory, this characteristic makes the ODP one of the largest public edited directories. However, with such a large editorial base, do irregularities or regularities arise within the directory structure? This project explores the use of Formal Concept Analysis (FCA) in understanding the structure of the ODP. Although the underlying driver of FCA is the ability to generate concept lattices from data, research in FCA has provided a variety of analysis tools. Initial analysis of generated concept lattices indicated that the ODP may be amenable to a horizontal decomposition. However, the sheer size of the directory made computing the entire concept lattice intractable. Thus, an alternate method was developed to perform a horizontal decomposition by computing only sections of the lattice. The techniques generated in this analysis can be applied to other domains, such as file system and software package structures.